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Abstract 

This article reports the development and validation study of tests to assess achievements at three levels of 
knowledge structure, following the model proposed by Sugrue to measure problem-solving skills. The literature 
has reported this proposal as a model consistent with the theoretical constructs underlying problem-based 
learning (PBL) methodology. The tests were constructed for a General Chemistry course in a curriculum of 
engineering, which implements PBL methodology at a Peruvian university. The content validation of the tests 
was performed, as well as a pilot implementation with Peruvian students of first year engineering. The results 
obtained in omissions percentage, difficulty degree, items response pattern and the point biserial coefficient 
(rpb), let us to conclude that these are appropriate tools for assessing these skills, mainly for the purpose of 
contributing to process facilitation as well as to future research in this line. 
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1 INTRODUCTION 

Despite the variety of approaches to problem-based learning, in terms of its definition and different modalities 
of implementation (Dochy, Segers, Van den Bossche & Gijbels, 2003; De Graaff & Kolmos, 2003), there is no 
doubt that there is a consensus to point out that from the cognitive dimension, one of the main objectives in 
this educational proposal is the development of problem solving skills both for the acquisition of knowledge, as 
for its application (Hmelo-Silver, 2004). Although there is a large volume of research on PBL, there is no 
consensus on what achievements or what measurement methods should be used to evaluate their effects, and 
this situation leads to the need to identify relevant theoretical constructs underlying PBL features and main 
goals in the learning context (Newman, 2003). The meta-analysis performed by Gijbels , Dochy , Van den 
Bossche and Segers (2005), used Sugrue's model (1994, 1995) as a frame of reference to study the PBL effects 
when the assessment of its main goals focuses on the understanding of concepts, principles that link concepts, 
and the linking of concepts and principles to conditions and procedures for application. For the authors, the 
great advantage of Sugrue model is that it is translated into specifications for the assessment of the main 
cognitive components of problem solving: knowledge structure, metacognitive functions, and motivation. 

This study follows the line of work of Gijbels et al. (2005), to develop instruments with the purpose to monitor 
student achievement in three levels of knowledge structure using Sugrue model, in a General Chemistry course 
where a hybrid PBL approach was implemented. 
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1.1 PBL theoretical framework 

PBL methodology is part of an educational vision that promotes an open, critical, and reflective learning with a 
holistic approach to knowledge that recognizes their complex and changing nature, and involves a community 
of people who interact collaboratively to make decisions regarding different problem situations they must face. 
PBL, in this sense, is the medium to set the conditions to promote active, contextualized, integrated and 
comprehension oriented learning, providing opportunities to reflect on the educational experience and to 
practice the application of learning (Margetson, 1997; Engel, 1997). Therefore, it is clear that the learning goals 
go far beyond the mere acquisition of disciplinary content. 

The scenario or problem that initiates the learning process fulfills several functions during the process: to 
organize the content and knowledge; provide the context of the learning environment, stimulate higher order 
thinking and reflection, as well as promoting and maintaining motivation for learning. Hung (2006) proposes an 
interesting model that clearly illustrates the different components of the scenario or problem design, and can 
be a reference to identify the main issues involved in PBL learning goals. 

The core components of the model: content, context and connection are mainly related to the ownership and 
adequacy of content knowledge, and their contextualization and integration. The content, as a central 
component, validates proper alignment between the problem scope and the curricular contents (expected 
learning) in breadth and depth. The context ensures that the situation presented is authentic (real or realistic), 
relevant to the professional area in which the student is formed, their personal interests or his life as a member 
of society. The connection component enables students to integrate knowledge and interconnect the concepts 
in such a way of establishing a conceptual network of the subject. 

Processing components: research, reasoning and reflection facilitate conscious and meaningful involvement of 
students in their learning process. They are the dynamic elements of the model, thus, research enables the core 
components, promotes the development of skills for the search and processing of information, facilitates 
conceptual connections for student learning and supports the processes of reasoning and reflection. The 
reasoning component promotes the development of skills of higher order thinking, actives core components 
also facilitates student conceptual connections and supports the processes of research and reflection. The latter 
component acts as a guiding element for metacognition, facilitates reflection process to synthesize and 
integrate the knowledge learned, promotes in students the habit to develop and use their skills to self-directed 
learning and lifelong learning. 

There is an additional important element to consider: PBL process is developed on the basis of efficient 
teamwork dynamics. The influence of interpersonal relationships and communication with others about 
learning is recognized in both learner-centered psychological principles proposed by APA (1997), and the 
constructivist view of teaching and learning (Coll, 2001). Working collaboratively with peers favors the 
activation of prior knowledge, provides opportunities for exchange of ideas contributing to the restructuring of 
patterns of knowledge, stimulates epistemic curiosity, attitude and motivation for learning (Schmidt, 1983). 

According to discussed so far, PBL methodology helps students to construct an extensive and flexible knowledge 
base; develop effective problem-solving skills; develop self-directed, lifelong learning skills; become effective 
collaborators; and become intrinsically motivated to learn (Hmelo-Silver, 2004). 

The adoption of PBL involves substantial changes in various aspects related to institution, teachers and 
students, which will ultimately determine the achievement of the fundamental goals of this educational 
proposal. This transition is not a simple process and therefore, it has emerged a variety of approaches to the 
methodology ranging from adoption in isolated courses belonging to a traditional curriculum, until joining a 
radical curricular transformation (Dochy et al., 2003). The diversity of approaches has hampered the 
interpretation of results of research on their effectiveness (Camp, 1996), which is why from the late nineties, 
several studies have recognized the need to consider the characteristics of the specific context in where the PBL 
implementation is performed, in order to have more and better information for evaluating the achievements of 
the methodology and the factors that may affect them (Dochy et al., 2003; Gijbels et al., 2005; Neville, 1999; 
Newman, 2003). 
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1.2 PBL approach implemented in the study 

In Latin America Engineering Programs the more frequent experiences correspond to PBL approaches, mainly 
hybrid models that incorporate different kind of variations of the original model, implemented in one single 
course. In the case of the General Chemistry course were this study was performed, each unit was organized as 
shown in Figure 1. 


• Presentation of the PBL scenario of the unit 

• First Brainstorm 

Autonomous work group 

k 

Learning activities carried out under the guidance 
of Professor 

• Group brainstorming and planning 
strategies to address the problem 

• Groups work in collaborative activities, 
designed by the professor and related to 
the learning objectives of the unit 

• Research and group discussion 

• Mediation of professor performs both in 
cognitive aspects, as well as in the group 
performance monitoring. 

• Review and adjustment of the 
strategy 

• Professor adapts the mediation, according 
to the progress of the students and 
stimulates the debate of ideas and 
decision-making. 

’ 

• End of the unit: groups present their proposals for a solution to the problem 


Figure 1. Hybrid PBL approach implemented in the General Chemistry course 


The assessment of each unit considers the proposal of solution for the PBL scenario, collaborative learning 
activities performed with mediation of the Professor, the electronic portfolio and two individual tests: one of 
multiple options to evaluate achievements in the three levels of the structure of knowledge according to the 
model of Sugrue, and other of open response relating to the contents of the unit. 

1.3 Problem solving skills in Sugrue model 

The literature on problem solving skills is characterized by displaying multiple theoretical frameworks from 
cognitive science or information processing. However, comprehensive models of its components have been 
proposed based on the review and compilation of results from several research lines. Sugrue examined 
common issues of some of these models, in order to identify a set of cognitive components that could be 
measured to estimate the extent to which a student can solve problems within a domain, such as science, as 
well as provide a basis for selecting a subset of variables to be submitted for evaluation. 

The three cognitive components to be assessed in problem solving following Sugrue model are: 

• Cognitive Functions, component that supports flexible adaptation of self-knowledge to meet the 
demands of a new problem. It is related to the so-called metacognitive functions, or processes of 
higher order thinking, 

• Beliefs, component that aims to generate a comprehensive profile of the student's ability and 
willingness to solve problems in a particular domain, through variables: self-efficacy , perception of 
task demands and perceived attractiveness of the task, 

• knowledge structure, this component consists of three levels that, in a perfect performance, show a 
high degree of connectivity, integration and consistency. 












Journal of Technology and Science Education - http://dx.doi.org/10.3926/jotse.100 


Sugrue suggests that the ability to solve problems in a particular domain results from the complex interaction of 
the structure of knowledge, metacognitive functions and motivation. The differences observed during the 
process, from the interpretation of the problem to the persistence in trying to solve it, can be attributed to 
variations in these three aspects of cognitive constructs. For each of the three categories of cognitive 
components, Sugrue describes a limited set of variables that should be targeted by assessment under two 
criteria: they were shown to be critical by research or open to instructional intervention. In this study, interest 
has focused on developing tools to assess achievements in the component knowledge structure, so this theme 
is developed in more detail in the next section. 

1.4 Knowledge structure assessment 

This component of Sugrue model considers three levels: 

• First level: Concepts. A concept is a category of objects, events, people, symbols and ideas that share 
common attributes and properties, and are identified by the same name. The evaluation of the 
understanding of a concept implies: selecting examples of the concept (in a multiple choice format), or 
generating of examples of the concept (in an open-ended format), or explanation of why examples 
reflect concept attributes (in a hands-on format). 

• Second level: Principles. A principle is defined as a rule, law, formula or statement that characterizes 
the relationship (often causal) between two or more concepts. The evaluation of the understanding of 
principles involves: selecting best predictions or best explanation (in a multiple choice format), or 
generating predictions or solutions (in an open-ended format), or giving an explanation of an event or 
result (in a hands-on format). 

• Third level: Linking the concepts and principles to conditions and procedures for application. To 
facilitate the resolution of the problem, the concepts and principles should be linked to the conditions 
and procedures that facilitate their use in new situations. A procedure is a set of steps that can be 
done to achieve a goal. The conditions are aspects of the context that indicate the existence of an 
instance of a concept, or to indicate that a principle is operating or can be applied, or that a particular 
procedure is appropriate. Subjects with good performance in problem solving should be able to 
recognize situations where procedures can be performed to identify or generate instances of a concept 
and should be able to carry out these procedures exactly. Overall, they should be able to assemble a 
procedure based on a principle to construct a desired achievement in a new situation. The evaluation 
of this level involves: selecting correct procedure for identifying instances of a concept, or selecting 
most appropriate procedure to change the state of one concept by manipulating other (in a multiple- 
choice format); or generating a procedure for identifying instances of a concept, or generating a 
procedure to change the state of one concept by manipulating other (in a multiple-choice format); or 
performing procedures to identify instances of a concept, or performing a procedure to change the 
state of one concept by manipulating other (in a hands-on format). 

The diagnostic evaluation of problem solving skills should allow the identification of students who understand 
the concepts but not the principle that binds them, students who understand the principles and concepts but 
have no knowledge of the procedures for applying them, and students who are able to perform procedures 
correctly but do not know when it is appropriate their application. 

The findings of Gijbels et al. (2005) showed no significant differences in the first level, corresponding to the 
understanding of concepts, comparing the achievements reported in PBL implementations with conventional 
teaching methodology. However, in the second level, corresponding to the understanding of principles linking 
concepts, PBL showed significant superiority. There were not enough studies on the third level of the model, for 
this reason, previous research aimed to develop tests to assess achievements in the third level that were 
applied to freshmen engineering courses, in the context of General Chemistry and General Physics courses 
(Morales 2011a, 2011b). 

The aim of the present study is to develop and validate instruments that can be used for the assessment of 
cognitive achievement in the three levels of knowledge structure, based on a model that is consistent with the 
learning environment promoted by methodologies such as PBL. 
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2 METHODOLOGY 

2.1 Instruments construction 

The first task in the construction of the tests was to identify the concepts, principles and the conditions and 
procedures that were to be evaluated in the main thematic units of General Chemistry 2 course. To do this, it 
was conducted a content analysis of topics and issues in the curriculum, to detail the domain and sub-domain 
of interest and decide the level of assessment. Then the items were written taken as reference Sugrue model 
for evaluating three levels of Knowledge Structure: concepts, principles and link the concepts and principles to 
conditions and procedures for application, resulting in the construction of three tests, one for each topic: 
Thermodynamics, Kinetics and Chemical Equilibrium. The format used for all the tests was multiple choices. 

The items were formulated in such a way that for the first level, the student should identify examples of the 
concept. At the second level, the student should select the best explanation of a particular event and in the 
third level, the student must select the correct procedure to identify concepts in a given situation, or select the 
most appropriate procedure to change a concept status manipulating another concept. 

Some examples: 

• First level: 

The soldiers often bring prepared food placed in a closed plastic bag or aluminium foil, which can be 
heated by immersing it in boiling water or by placing it on the hot engine of a vehicle. Another 
possibility is to use the "flameless heaters," which utilize a reaction between magnesium metal and 
water, which is very exothermic. 

From the text above: 

Identify a thermodynamic system and indicate its type (open, closed, isolated), justifying your answer. 

• Second level: 

It is suspected that the decomposition of a biological substance follows a first order kinetics. To 
confirm this, the concentration of the substance was measured at different times. What must we do 
with this data? 

• We must determine the concentration which decomposed, as the difference between the 
initial concentration and the remaining concentration at each time. If the graph of the 
obtained values versus time is a straight line, then it is a first order reaction. 

• We must calculate the inverse of concentration at each time. If the graph of the obtained 
values versus time is a straight line, then it is a first order reaction. 

• We must determine the log of the inverse of concentration at each time. If the graph of the 
obtained values versus time is a straight line, then it is a first order reaction. 

• We must determine the log of the concentration at each time. If the graph of the obtained 
values versus time is a straight line, then it is a first order reaction. 

• Third level: 

We want to increase the acidity of a solution containing 0.1 M acetic acid (Ka = 1.8 x 10' 5 ). In the 
laboratory we find the following aqueous solutions: hydrocyanic acid 0.1 M (Ka = 6.2 x 10 10 ) and 
nitrous acid 0.1 M (Ka = 4.0 x 10' 4 ). Which of the following alternatives will achieve the goal? 

• Add a volume of the nitrous acid solution, which is a stronger acid and provide greater 
amount of H+ ions to the acetic acid solution, increasing its acidity. 

• Add a volume of the hydrocyanic acid solution, which is a stronger acid and provide greater 
amount of FI+ ions to the acetic acid solution, increasing its acidity. 

• Increase the volume of the acetic acid solution by adding water, so acid ionization will increase 
and therefore the acidity. 







Journal of Technology and Science Education - http://dx.doi.org/10.3926/jotse.100 


2.2 Content validation 

This is the most important stage in the validation process, as it shows the logical relationship between test 
items and content and skills assessed. For this case, five Chemistry professors from Pontifical Catholic University 
of Peru (PUCP) were invited to assume the role of judges in assessing the degree of coherence between the 
concepts and principles to be assessed with each item and the item statement. Also, they were asked to assess 
the degree of consistency between the learning objectives associated with each item and the item statement. 

On the basis of the judges' evaluation and their comments and suggestions, adjustments were made to the 
items in each test, to proceed with their pilot application and a reassessment by three Chemistry professors 
from PUCP who participated as judges. 

2.3 Pilot application 

The tests were applied with students of General Chemistry 2 course (PUCP) following a hybrid PBL approach in 
different semesters between 2010 and 2011. The ages of the students involved ranged between 17 and 20 
years, with the highest percentage at the age of 17 years (43.2%). Table 1 reports further information about 
participants in the pilot implementation of each test. 


Test 

N° participants (N) 

Gender (%) 

Male 

Female 

Thermodynamics 

328 

73,1 

26,9 

Kinetics 

313 

69,0 

31,0 

Chemical Equilibrium 

283 

70,5 

29,5 


Table 1. Information about participants in the pilot implementation of each test 


The test application was always made at the end of the corresponding thematic unity. The maximum duration 
of application was 50 min. With the tests responses, it was performed the analysis of items for each of the 
three tests. For this purpose it was determined: 

• the percentage of omissions, corresponding to reactive responses that have been left in blank, 

• the difficulty degree, which is a measure of the number of subjects who responded correctly to the 
reagent; 

• biserial point coefficient (rpb), which is a special case of Pearson correlation for determining the 
correlation between the reactant and the total score and constitutes an internal discrimination index. 

3 RESULTS 

3.1 Instruments construction 

Table 2 summarizes the final structure of each test. 


Test 

N° First level 

items 

N° Second level items 

N° Third level items 

Thermodynamics 

2 

1 

3 

Kinetics 

2 

2 

2 

Chemical Equilibrium 

1 

1 

3 


Table 2. Final structure of each test 
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3.2 Content validation 

Table 3 summarizes the results of the second judges' evaluation of the built tests. The evaluation by the second 
group of judges was satisfactory, qualifying the degree of consistency between the questions and the learning 
objectives, concepts and principles associated, as good or very good. 


Test 

Degree of consistency (%, 

between test items and: 

Learning objectives 

Concepts and principles 

Very good 

Good 

Very good 

Good 

Thermodynamics 

78 

22 

83 

17 

Kinetics 

83 

17 

89 

11 

Chemical Equilibrium 

80 

20 

87 

13 


Table 3. Summary of second judges' evaluation of the built tests 


3.3 Pilot application 

Table 4 summarizes the results of analysis of items for each of the three tests. The percentage of omissions is 
not included, because in all cases the participating students answered the total of questions. 


Test 

Level 

Item 

Difficulty 
degree (%) 

Proportion of 

errors 

rpb 

Thermodynamics 

1 

1 

56,4 

0,436 

0,433 

1 

2 

65,5 

0,345 

0,475 

2 

3 

54,6 

0,454 

0,585 

3 

4 

41,5 

0,585 

0,478 

3 

5 

80,8 

0,192 

0,418 

3 

6 

43,6 

0,564 

0,466 

Kinetics 

1 

1 

66,8 

0,332 

0,488 

1 

2 

83,7 

0,163 

0,476 

2 

3 

78,9 

0,211 

0,461 

2 

4 

62,9 

0,371 

0,484 

3 

5 

64,2 

0,358 

0,511 

3 

6 

57,4 

0,426 

0,469 

Chemical 

Equilibrium 

1 

1 

71 

0,29 

0,510 

2 

2 

35,3 

0,647 

0,519 

3 

3 

56,2 

0,438 

0,586 

3 

4 

32,2 

0,678 

0,463 

3 

5 

61,5 

0,385 

0,476 


Table 4. Summary of the results of analysis of items for each test 


The average of degree of difficulty for the Thermodynamics test was 57.07%; for Kinetics test the value 
obtained was 68.98% and for Chemical Equilibrium test was 51.24%. According to this, the second test can be 
classified as moderately difficult, while the first and the third tests would qualify as medium difficulty. The 
variability of the values obtained for the items in each test was adequate, showing that it contained questions 
of different levels of difficulty for students. Biserial point coefficient (rpb) is located in the range 0.418 to 0.586 
for all items of the tests, so it is an indication that the reagents of each test discriminate between subjects 
achieving high score on the test and those with low scores. 
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Figure 2 shows, as an example, a comparison of the results in the evaluation of achievements in the three levels 
of knowledge structure of three students, using Thermodynamics test. 



Figure 2. Evaluation results on the three levels of knowledge 
structure of three students in a General Chemistry course 


It can be seen that the student RBS had very satisfactory results according to the learning goals of PBL 
methodology. The student BBG had very good levels of achievement in concepts and principles, but must 
improve his skills in the third level. Something similar happens with the student DRI, although his results are 
lower, especially in the third level, so he may require special attention. 

The overall results in the application of these instruments with a group of students allow the professor to get 
information about achievement in terms of expectations in line with PBL approach. According to it, the 
facilitator can have evidence to reformulate the problems, learning activities or his facilitation. 

For example, Figure 3 shows that the results in Chemical Equilibrium test were lowest in the group of students 
assessed, so it would be necessary to analyze the reasons of this situation for planning corrective action. 



Figure 3. Global results on the three levels of knowledge 
structure for each test 


4 CONCLUSIONS 

The results obtained in the validation of the built instruments are highly satisfactory, since, first, ensure a high 
degree of consistency between the learning objectives, the content assessed and the items in each test. The 
theoretical foundations of the model have been applied rigorously and are relevant to the learning contexts 
promoted by PBL methodology. 
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In relation to the analysis of items in each of the tests, it can be seen that a very good range is reached in the 
coefficient of internal discrimination, which is very valuable for future studies using these instruments. It can be 
corroborated also a trend toward higher levels of difficulty in items of the third level of knowledge structure. It 
is noteworthy that in PBL implementations are expected to have significant achievements especially at this 
level. 

As noted in earlier sections of this report, there is a major difficulty for the evaluation of achievements in the 
implementation of PBL methodology, which is related to the construction and application of appropriate tools 
for measurement. Sugrue's model provides a suitable and understandable framework that supports the criteria 
for the construction of test to assess problem-solving skills, such as demonstrated in our investigations. In this 
study, the model helped the development of three instruments to assess achievements in three basic thematic 
units of General Chemistry. According to the results of the validation, the tests built have robust features for 
their use as tools to monitor student achievement in three levels of knowledge structure using Sugrue model, 
as well as in future research to explore achievements in the use of PBL methodology in contexts similar to that 
reported here. 
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